<html>
<head><meta charset="utf-8"><title>internals post · t-compiler/wg-pipelining · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/index.html">t-compiler/wg-pipelining</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html">internals post</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="165913206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165913206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165913206">(May 17 2019 at 16:08)</a>:</h4>
<p>Cargo support is now in nightly! I'm drafting up an internals post and will post it here soon</p>



<a name="165913277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165913277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165913277">(May 17 2019 at 16:09)</a>:</h4>
<p>Ok, <a href="https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199" target="_blank" title="https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199">posted</a>!</p>



<a name="165914372"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165914372" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165914372">(May 17 2019 at 16:24)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> rustup update only gets me 2019-05-16</p>



<a name="165914401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165914401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165914401">(May 17 2019 at 16:25)</a>:</h4>
<p>oh I should probably say that</p>



<a name="165914406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165914406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165914406">(May 17 2019 at 16:25)</a>:</h4>
<p>I just guessed</p>



<a name="165914426"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165914426" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165914426">(May 17 2019 at 16:25)</a>:</h4>
<p>? so 16 has pipelining?</p>



<a name="165914682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165914682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165914682">(May 17 2019 at 16:28)</a>:</h4>
<p><code>rustc 1.36.0-nightly (7d5aa4332 2019-05-16)</code> that has pipelining</p>



<a name="165914696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165914696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165914696">(May 17 2019 at 16:28)</a>:</h4>
<p>I think it's <code>rustup update nightly-2019-05-17</code></p>



<a name="165916106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165916106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165916106">(May 17 2019 at 16:47)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> with lark that just made pipelined compilation a sealed and done deal for me</p>



<a name="165916124"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165916124" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165916124">(May 17 2019 at 16:47)</a>:</h4>
<p>those numbers are "this is what rustc devs will see once we switch everything to rlibs"</p>



<a name="165916126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165916126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165916126">(May 17 2019 at 16:47)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> I was surprised</p>



<a name="165933186"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165933186" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165933186">(May 17 2019 at 20:19)</a>:</h4>
<p>I'm collecting all the results in a spreadsheet here - <a href="https://docs.google.com/spreadsheets/d/1CU7o3IocPtNAUevvrPsTI77z_AexsRloUgiusiXJWtg/edit?usp=sharing" target="_blank" title="https://docs.google.com/spreadsheets/d/1CU7o3IocPtNAUevvrPsTI77z_AexsRloUgiusiXJWtg/edit?usp=sharing">https://docs.google.com/spreadsheets/d/1CU7o3IocPtNAUevvrPsTI77z_AexsRloUgiusiXJWtg/edit?usp=sharing</a></p>



<a name="165933188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165933188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165933188">(May 17 2019 at 20:19)</a>:</h4>
<p>from the internals post</p>



<a name="165933327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165933327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165933327">(May 17 2019 at 20:21)</a>:</h4>
<p>if you want you can make the google spreadsheet coloring be gradient which .. could be helpful? (i.e., from light green for low % wins to darker for high % wins)</p>



<a name="165933439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165933439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165933439">(May 17 2019 at 20:22)</a>:</h4>
<p>oh nice</p>



<a name="165933444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165933444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165933444">(May 17 2019 at 20:22)</a>:</h4>
<p>/me pokes</p>



<a name="165933778"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165933778" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165933778">(May 17 2019 at 20:27)</a>:</h4>
<p>nice, that does look much better</p>



<a name="165936641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165936641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165936641">(May 17 2019 at 21:03)</a>:</h4>
<p>So an interesting result I'm seeing is that this feels like some compelling data that we want to do the next step of pipelining</p>



<a name="165936649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165936649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165936649">(May 17 2019 at 21:03)</a>:</h4>
<p>which is that cargo should be able to pipeline compliations that end in the linker</p>



<a name="165936667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165936667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165936667">(May 17 2019 at 21:04)</a>:</h4>
<p>(e.g. compiling a binary or a proc-macro)</p>



<a name="165936714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165936714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165936714">(May 17 2019 at 21:04)</a>:</h4>
<p>projects can always be reorganized to have a tiny crate which is the binary which basically just serves the purpose of calling the linker</p>



<a name="165936721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165936721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165936721">(May 17 2019 at 21:04)</a>:</h4>
<p>but it's a bummer to have to do that manually</p>



<a name="165936732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165936732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165936732">(May 17 2019 at 21:04)</a>:</h4>
<p>although this is also the same w/ sccache where rlibs are just inherently easier to work with than linked artifacts</p>



<a name="165937173"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165937173" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165937173">(May 17 2019 at 21:10)</a>:</h4>
<p>fwiw I think there's already a general desire/feeling in the community to do the tiny binary split and big library backing</p>



<a name="165937197"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165937197" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165937197">(May 17 2019 at 21:11)</a>:</h4>
<p>so that might be somewhat low priority -- I wonder how difficult it would be to get stats on that</p>



<a name="165937212"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165937212" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165937212">(May 17 2019 at 21:11)</a>:</h4>
<p>maybe lines of code in <a href="http://main.rs" target="_blank" title="http://main.rs">main.rs</a> after expansion (i.e., including modules) and <a href="http://lib.rs" target="_blank" title="http://lib.rs">lib.rs</a> or something along those lines</p>



<a name="165938590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165938590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165938590">(May 17 2019 at 21:32)</a>:</h4>
<p>that's true yeah, a bunch of stuff is already architected this way but I know of big stuff which isn't at least</p>



<a name="165938595"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165938595" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165938595">(May 17 2019 at 21:32)</a>:</h4>
<p>e.g. unit tests can take awhile to compile, cargo's binary is relatively big, sccache's is huge,e tc</p>



<a name="165938604"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165938604" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165938604">(May 17 2019 at 21:33)</a>:</h4>
<p>same w/ serde_derive</p>



<a name="165938998"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165938998" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165938998">(May 17 2019 at 21:39)</a>:</h4>
<p>yeah, I think proc macros would probably benefit quite a bit from that layout</p>



<a name="165939027"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165939027" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165939027">(May 17 2019 at 21:40)</a>:</h4>
<p>since generally they're written inline, somewhat unlike binary/library split since proc macro internals aren't really public, unlike lots of binary internals</p>



<a name="165988303"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165988303" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165988303">(May 18 2019 at 19:42)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> you typoed the environment variable in one place in your post and it's causing chaos lol</p>



<a name="165989047"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/165989047" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#165989047">(May 18 2019 at 20:03)</a>:</h4>
<p>oopsie</p>



<a name="166091074"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166091074" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166091074">(May 20 2019 at 14:49)</a>:</h4>
<p>So continuing to collate data, it looks like pipelining across the board is not a regression on any front, it averages a 10% reduction in build time, and we're seeing up to twice as fast builds</p>



<a name="166123896"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166123896" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166123896">(May 20 2019 at 22:09)</a>:</h4>
<p>I've opened a <a href="https://github.com/rust-lang/rust/issues/60988" target="_blank" title="https://github.com/rust-lang/rust/issues/60988">dedicated tracking issue now</a> for stabilizing pipelined compilation given how compelling the data is</p>



<a name="166125908"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166125908" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166125908">(May 20 2019 at 22:41)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> Thanks for the measurements and internals post. Initial results are promising, though the improvements are not as high/universal as I had hoped.</p>



<a name="166125980"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166125980" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166125980">(May 20 2019 at 22:42)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> But I can't manage to get the 2019-05-17 Cargo to test. <code>rustup update nightly</code> gives me 2019-05-15, as does <code>rustup update nightly-2019-05-17</code>. What am I missing?</p>



<a name="166126025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166126025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166126025">(May 20 2019 at 22:43)</a>:</h4>
<p>my rustc is 2019-05-19 after <code>rustup update nightly</code>, but the cargo is older</p>



<a name="166126139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166126139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166126139">(May 20 2019 at 22:45)</a>:</h4>
<p><span class="user-mention" data-user-id="120989">@nnethercote</span> Were all the non-results on small numbers of cores? I think its reasonable to expect that Cargo + codegen-units is pretty good at keeping a small number of cores busy. The main wins will be when there's currently a lot of idle cores.</p>



<a name="166126162"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166126162" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166126162">(May 20 2019 at 22:45)</a>:</h4>
<p>I don't know. I have a 14-physical-core machine, I plan to test the full rustc-perf benchmark (~30 programs) once I can</p>



<a name="166127231"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127231" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127231">(May 20 2019 at 23:03)</a>:</h4>
<p><span class="user-mention" data-user-id="120989">@nnethercote</span> if you <code>rustup update nightly</code> it should be good enough</p>



<a name="166127237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127237">(May 20 2019 at 23:03)</a>:</h4>
<p>the commit/date reported by <code>-V</code> is the commit date not the build date</p>



<a name="166127288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127288">(May 20 2019 at 23:04)</a>:</h4>
<p><span class="user-mention" data-user-id="120989">@nnethercote</span> I agree it's not quite as good as I hoped, but still well within the range of "worth the stabilization effort"</p>



<a name="166127296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127296">(May 20 2019 at 23:04)</a>:</h4>
<p>I think there's actually even more to be gained by going a step further and pipelining up to the linker</p>



<a name="166127300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127300">(May 20 2019 at 23:04)</a>:</h4>
<p>but we'd want to gather more data first</p>



<a name="166127322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127322">(May 20 2019 at 23:05)</a>:</h4>
<p>it's also worth keeping in mind what I mentioned in the first post on the thread that incremental builds are likely to see a much bigger benefit than whole crate builds</p>



<a name="166127330"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127330" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127330">(May 20 2019 at 23:05)</a>:</h4>
<p>but everyone's only been measuring whole crate builds</p>



<a name="166127333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127333">(May 20 2019 at 23:05)</a>:</h4>
<p>which makes the most sense of course because it's the easiest thing to do</p>



<a name="166127336"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127336" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127336">(May 20 2019 at 23:05)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> oh, ok, thanks for the clarification. I definitely agree it's worth stabilizing, and there may be room for more improvement, e.g. by generating metadata in parallel with type-checking/borrow-checking.</p>



<a name="166127512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127512">(May 20 2019 at 23:08)</a>:</h4>
<p><span class="user-mention" data-user-id="120989">@nnethercote</span> I'd sort of love to get to a world where Cargo can spawn literally all rustc instances for a crate graph immediately, and then rustc just queries cargo every now and then for "wake me up when this is ready" or "this is ready" and Cargo orchestrates the notifications</p>



<a name="166127544"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127544" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127544">(May 20 2019 at 23:09)</a>:</h4>
<p>like I could imagine rustc being a black box to Cargo and it just says "needs Vec&lt;String&gt; or "produced String" and Cargo just wakes things up as necessary assuming all the strings are unique and all</p>



<a name="166127557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127557">(May 20 2019 at 23:09)</a>:</h4>
<p>that way we could at least parse the whole crate graph in parallel</p>



<a name="166127560"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127560" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127560">(May 20 2019 at 23:09)</a>:</h4>
<p>would of course be much more difficult to implement :)</p>



<a name="166127625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127625">(May 20 2019 at 23:10)</a>:</h4>
<p>I guess I'm also thinking that like if a crate requires a procedural macro it actually can progress largely through the whole resolution phase of the compiler until it finally needs the macro</p>



<a name="166127629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127629">(May 20 2019 at 23:10)</a>:</h4>
<p>I dunno, these may all be small wins</p>



<a name="166127942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166127942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166127942">(May 20 2019 at 23:16)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> All interesting ideas! Anyway, I should be pleased with the currently results, given that they are quite good for what is the absolute simplest implementation :)  I will do the rustc-perf measurements over the next couple of days, including incremental. I'll do Firefox as well.</p>



<a name="166128533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166128533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166128533">(May 20 2019 at 23:25)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> I've been thinking about a model where a graph orchestration engine can start a graph of compilers and then start feeding them incremental input at the keystroke level (ie, rather than linking cargo into rls, make it cheap enough for rls to invoke "real" builds). That would require lots of fine-grained control over what outputs you want from each compilation step, which perhaps changes dynamically.</p>



<a name="166128600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166128600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166128600">(May 20 2019 at 23:26)</a>:</h4>
<p><span class="user-mention" data-user-id="213049">@Jeremy Fitzhardinge</span> that does indeed sound like the dream :)</p>



<a name="166128613"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166128613" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166128613">(May 20 2019 at 23:26)</a>:</h4>
<p>jturner was talking about that at the last all-hands</p>



<a name="166128614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166128614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166128614">(May 20 2019 at 23:26)</a>:</h4>
<p>certainly an ambitious goal :)</p>



<a name="166128841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166128841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166128841">(May 20 2019 at 23:30)</a>:</h4>
<p>I'm very interesting in continuing the conversation about how Buck and Cargo can work together. My project for the back half of the year will be trying to auto-generate buck build rules from Cargo.toml for <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a>, and there's a number of improvements to Cargo's model I'd like to discuss in that context.</p>



<a name="166138354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138354">(May 21 2019 at 03:08)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> wait, what is this pipelining up to if not the linker?</p>



<a name="166138454"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138454" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138454">(May 21 2019 at 03:11)</a>:</h4>
<p><span class="user-mention" data-user-id="137587">@Gankro</span> If I understood you correctly, then we currently pipeline split after metadata (i.e., same output as you get from cargo check) is done but LLVM and linker has not yet run</p>



<a name="166138516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138516">(May 21 2019 at 03:12)</a>:</h4>
<p>so this is proposing splitting the llvm+link step into two parts? What could we possibly execute only once llvm was done, but before linking is done?</p>



<a name="166138584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138584">(May 21 2019 at 03:14)</a>:</h4>
<p>er, not sure what you mean -- the stable/no flags cargo has no pipelining</p>



<a name="166138590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138590">(May 21 2019 at 03:15)</a>:</h4>
<p>(maybe I wasn't clear)</p>



<a name="166138597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138597">(May 21 2019 at 03:15)</a>:</h4>
<p>i.e., we always wait for LLVM and linking to finish if we're running it (debug/release builds)</p>



<a name="166138599"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138599" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138599">(May 21 2019 at 03:15)</a>:</h4>
<p>alex said "I think there's actually even more to be gained by going a step further and pipelining up to the linker"</p>



<a name="166138603"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138603" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138603">(May 21 2019 at 03:16)</a>:</h4>
<p>I believe that might be referring to us not pipelining the final binary -- which can often be quite large</p>



<a name="166138647"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138647" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138647">(May 21 2019 at 03:16)</a>:</h4>
<p>but not sure</p>



<a name="166138655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138655">(May 21 2019 at 03:16)</a>:</h4>
<p>I'm also .. not entirely sure what that would mean</p>



<a name="166138670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138670">(May 21 2019 at 03:17)</a>:</h4>
<p>oh right, the diagram does say we wait for all our libs to be fully codegen'd before even starting to compile the binary, which is odd yeah</p>



<a name="166138671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166138671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166138671">(May 21 2019 at 03:17)</a>:</h4>
<p>but I guess in theory we could run LLVM before dependency LLVMs finish?</p>



<a name="166173836"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166173836" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166173836">(May 21 2019 at 13:51)</a>:</h4>
<p><span class="user-mention" data-user-id="137587">@Gankro</span> yeah so the method of pipelining is pretty course and selective now, only when an rlib depends on another rlib can we pipeline those two compilations</p>



<a name="166173852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166173852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166173852">(May 21 2019 at 13:51)</a>:</h4>
<p>we, for example, can't pipeline an executable depending on a bunch of other rlibs</p>



<a name="166173860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166173860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166173860">(May 21 2019 at 13:51)</a>:</h4>
<p>(or any linked artifact for that matter)</p>



<a name="166173872"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166173872" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166173872">(May 21 2019 at 13:51)</a>:</h4>
<p>similarly we can't pipeline anything depending on a build script or a procedural macro</p>



<a name="166173886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166173886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166173886">(May 21 2019 at 13:51)</a>:</h4>
<p>so the general idea is just enabling more paralellism by having better synchronization between rustc/cargo</p>



<a name="166174001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166174001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166174001">(May 21 2019 at 13:53)</a>:</h4>
<p>cool</p>



<a name="166337317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166337317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Marco <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166337317">(May 23 2019 at 06:54)</a>:</h4>
<p>Why do the compiler invocations in <a href="https://github.com/rust-lang/compiler-team/blob/master/working-groups/pipelining/NOTES.md#step-2-work-with-only-metadata-as-input" target="_blank" title="https://github.com/rust-lang/compiler-team/blob/master/working-groups/pipelining/NOTES.md#step-2-work-with-only-metadata-as-input">https://github.com/rust-lang/compiler-team/blob/master/working-groups/pipelining/NOTES.md#step-2-work-with-only-metadata-as-input</a> require <code>rustc libA.rs --emit metadata,link --crate-type lib</code> instead of <code>rustc libA.rs --emit metadata --crate-type lib</code>? <br>
I can see the the <code>libA.rmeta</code>produced is different <em>sometimes</em>, but that's surprising.</p>
<p>Fiddling with it locally and run into an ICE w/ the latter:</p>
<div class="codehilite"><pre><span></span>&gt;&gt;  rustc a.rs --emit=metadata --crate-type=lib
&gt;&gt; rustc b.rs --emit=link --crate-type=rlib --extern=a=liba.rmeta
error: internal compiler error: src/librustc_mir/monomorphize/collector.rs:775: Cannot create local mono-item for DefId(13:12 ~ a[8787]::number[0])

thread &#39;rustc&#39; panicked at &#39;Box&lt;Any&gt;&#39;, src/librustc_errors/lib.rs:637:9
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: aborting due to previous error


note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.36.0-nightly (37ff5d388 2019-05-22) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type rlib
</pre></div>



<a name="166337391"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166337391" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Marco <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166337391">(May 23 2019 at 06:56)</a>:</h4>
<p>I was looking to sketch the invoke-rustc-twice approach in Bazel, but ran into that immediately</p>



<a name="166378472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166378472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166378472">(May 23 2019 at 16:47)</a>:</h4>
<p><span class="user-mention" data-user-id="222771">@Marco</span> currently in rustc metadat is different if you emit just metadata vs if you emit metadata + a lib</p>



<a name="166378478"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166378478" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166378478">(May 23 2019 at 16:47)</a>:</h4>
<p>we also want to use one rustc process to emit both</p>



<a name="166378494"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166378494" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166378494">(May 23 2019 at 16:47)</a>:</h4>
<p>( so we don't need one rustc process for metadata and one for the lib)</p>



<a name="166382326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166382326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Marco <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166382326">(May 23 2019 at 17:36)</a>:</h4>
<p>Is it intentionally different? Does it make sense that it's non-deterministically different?</p>
<p>I wanted to see if the redundant metadata generation work in doing 2 rustc invocations from bazel is still a benefit, since that is substantially simpler to implement.</p>



<a name="166421627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166421627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166421627">(May 24 2019 at 04:14)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> I did some measurements: <a href="https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199/62?u=nnethercote" target="_blank" title="https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199/62?u=nnethercote">https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199/62?u=nnethercote</a></p>



<a name="166456713"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166456713" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166456713">(May 24 2019 at 14:17)</a>:</h4>
<p>Wow <span class="user-mention" data-user-id="120989">@nnethercote</span> that's quite comprehensive! Thanks for gathering all that!</p>



<a name="166456726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166456726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166456726">(May 24 2019 at 14:17)</a>:</h4>
<p>FWIW rustc sees no benefit b/c it has no pipelining opportunities</p>



<a name="166456796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166456796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166456796">(May 24 2019 at 14:18)</a>:</h4>
<p>and I think in general that's why pipelining isn't as great as we expected is that there's just fewere pipelining opportunities than we originally though</p>



<a name="166490339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166490339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166490339">(May 24 2019 at 21:07)</a>:</h4>
<p><span class="user-mention" data-user-id="222771">@Marco</span> I also want to prototype pipelining with separate invocations, so it would be useful to be able to do <code>--emit metadata</code> which generates the same .rmetas as <code>--emit metadata,rlib</code>. I wonder if <code>-Zalways-encode-mir</code> makes up the difference?</p>



<a name="166492691"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166492691" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166492691">(May 24 2019 at 21:37)</a>:</h4>
<p><span class="user-mention" data-user-id="213049">@Jeremy Fitzhardinge</span> I don't think <code>-Zalways-encode-mir</code> will be enough (you can read more about that at <a href="https://github.com/rust-lang/rust/issues/58465#issuecomment-479032740" target="_blank" title="https://github.com/rust-lang/rust/issues/58465#issuecomment-479032740">https://github.com/rust-lang/rust/issues/58465#issuecomment-479032740</a>), unless it has been changed recently.  I also would expect running the compiler twice would be substantially slower, since the second invocation would need to repeat all the work of the first (unless you intend to do incremental everywhere?).  On average, the codegen portion only covers about 30% of compile time, so it would be repeating the first 70% for every crate.</p>



<a name="166499037"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166499037" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Marco <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166499037">(May 24 2019 at 23:37)</a>:</h4>
<p>The thread doesn't say whether <code>-Zalways-encode-mir</code> suffices or not. Running the compiler twice should still enable a speedup if there's unused parallelism (and it is much easier to implement/experiment with in Bazel this way), but it's not strictly better like the alternative. </p>
<p>Has changing compilation unit to module instead of crate come up in this vein? In the abstract (ie. in ignorance) it seems like a similar way to make more pipelining possible.</p>



<a name="166595406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166595406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166595406">(May 26 2019 at 22:35)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> why does rustc have no pipelining opportunities? librustc takes so long that I was hoping to get some speedup there.</p>



<a name="166599658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166599658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166599658">(May 27 2019 at 00:42)</a>:</h4>
<p>The time is spent in LLVM, which is parallel already. Might help with incremental when few LLVM modules change</p>



<a name="166711001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166711001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166711001">(May 28 2019 at 12:31)</a>:</h4>
<p><span class="user-mention" data-user-id="120989">@nnethercote</span> rustc is entirely dylibs right now, which means that there are no pipelining opportunities</p>



<a name="166711024"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166711024" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166711024">(May 28 2019 at 12:31)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/pull/59800" target="_blank" title="https://github.com/rust-lang/rust/pull/59800">https://github.com/rust-lang/rust/pull/59800</a> is the solution for that</p>



<a name="166766597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166766597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166766597">(May 28 2019 at 22:39)</a>:</h4>
<p>I see, thanks for the info</p>



<a name="166932309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166932309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Marco <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166932309">(May 30 2019 at 18:13)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> (asking this again:) Has changing compilation unit to module instead of crate come up in this vein? In the abstract (ie. in ignorance) it seems like a similar way to make more pipelining possible.</p>



<a name="166945589"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166945589" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166945589">(May 30 2019 at 20:59)</a>:</h4>
<p><span class="user-mention" data-user-id="222771">@Marco</span> oh oops sorry must have missed this earlier! Currently we haven't considered that, mostly because it doesn't really fit cleanly into the compilation model today and would be a pretty significant undertaking</p>



<a name="166945606"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166945606" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166945606">(May 30 2019 at 20:59)</a>:</h4>
<p>the current implementation of pipelining was actually very low effort for what is hoped to be quite a high reward (relative)</p>



<a name="166945632"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166945632" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166945632">(May 30 2019 at 20:59)</a>:</h4>
<p>but you're right in that it's by no means the end-all-be-all of pipelining compilations, and there's a lot of theoretical possibilities for how we can improve things even more</p>



<a name="166945647"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166945647" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166945647">(May 30 2019 at 20:59)</a>:</h4>
<p>it's mostly just a question of balancing that with the amoutn of effort needed to implement</p>



<a name="166950533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166950533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Marco <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166950533">(May 30 2019 at 22:13)</a>:</h4>
<p>Got it. Is there anywhere that has listed out some of those possibilities?</p>



<a name="166999373"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/internals%20post/near/166999373" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/internals.20post.html#166999373">(May 31 2019 at 14:02)</a>:</h4>
<p><span class="user-mention" data-user-id="222771">@Marco</span> not currently, but we should probably make one!</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>